Cognitive Modeling with Context Sensitive Reinforcement Learning

A reinforcement learning system is typically described as a black box which receives two types of input, the current state, S, and the current reinforcement, R. From these two inputs, the system has to figure out a policy that determines what action to perform in each state to maximize the received reinforcement in the future (Sutton & Barto, 1998). The future expected reinforcement can be estimated either by using the sum of all future reinforcement or with an exponentially decaying time horizon. It is also possible to only take into account the reinforcement received at the next goal action which results in finite horizon algorithms (e. g. Balkenius & Morén, 1999). Learning is viewed as the formation of assocations between states and actions and are represented by numerical values that are changed during learning. In most basic reinforcement learning algorithm, the policy for each state is learned individually without regard for the similarity between different states. It would obviously be valuable if actions learned in one state could be generalized to other similar states. Such generalization can be introduced into a reinforcement learning algorithm in several ways. One possibility is to code the similarity between states by similar state vectors. Such methods have been proposed by Sutton (1996), who used a tile representation or the underlying state space and Balkenius (1996), who used a multi-resolution representation. As alternative is to learn the underlying state representation during exploration based on the closeness of different states (Dayan, 1993). In both cases, learning becomes faster since each learning instance will be generalized to many similar states. In many cases, it makes sense to divide the state input into two parts, one that code for the situation or context and one that codes for the part of the state that controls the action (cf. Balkenius & Hulth, 1999, Houghes & Drogoul, 2001). If such a combined representation is used together with the reinforcement algorithms described above, learning will generalize not only to similar states but also to similar contexts. The role of state and context will thus be symmetric.